feat: add readiness and liveness probes to crossplane #4748
Conversation
Signed-off-by: Philippe Scorsolini <p.scorsolini@gmail.com>
| return errors.Wrap(err, "cannot create ping health check") | ||
| } | ||
|
|
||
| // Add probes waiting for the webhook server if webhooks are enabled |
There was a problem hiding this comment.
Is there a way to add something like a started checker for the controllers?
Could it be helpful to report as ready only after controllers are operational, e.g. after acquiring the lease successfully.
There was a problem hiding this comment.
We could also check/do that only after we're aware of a use case that it would help.
Feel free to ignore for now.
There was a problem hiding this comment.
Yeah, we could definitely do something, but I couldn't find many examples of other projects doing much more than this. Acquiring the lease could not be the right condition though as we want to support multiple replicas serving webhooks, @sttts was talking about caches being populated, any additional hint/pointer to prior art?
There was a problem hiding this comment.
I don't think readiness is connected to leader election. It's a blackbox property whether the pod is ready to serve traffic (via the webhook). Controllers are always async. No need to wait for them. What does waiting for them even mean?
There was a problem hiding this comment.
If functionality of the process depends on informers to sync, then they should be part of readiness. E.g. admission in kube cannot work correctly without synced informers. Hence, the kube-apiserver might return inconsistent results prior to that.
Don't think it matters here much.
You could delay readiness by mgr.GetCache().WaitForSync() or so.
Signed-off-by: Philippe Scorsolini <p.scorsolini@gmail.com>
|
Hi, I still didn't understood how fix the error Internal error occurred: failed calling webhook "compositions.apiextensions.crossplane.io": failed to call webhook: Post "https://crossplane-webhooks.crossplane-system.svc:9443/validate-apiextensions-crossplane-io-v1-composition?timeout=10s": context deadline exceeded Just to make sure endpoint is responding: / # curl https://crossplane-webhooks.crossplane-system:9443/validate-apiextensio curl failed to verify the legitimacy of the server and therefore could not What I need to do to fix it? Thanks |
|
@diegoavitorino please open an issue with more details about your setup. |
Description of your changes
Fixes part of E2E flakiness we are observing, avoiding Crossplane to be marked as Ready before the Webhook server is actually ready. If webhook is not enabled (
webhooks.enabled=false) both probes are just set to the default no-op.I have:
Added or updated unit and E2E tests for my change.make reviewableto ensure this PR is ready for review.Addedbackport release-x.ylabels to auto-backport this PR, if necessary.Opened a PR updating the docs, if necessary.